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Abstract: The ship motion perception approaches mainly use maritime radar, Automatic Identification 
System (AIS) and cameras. However, using either of these approaches alone may result in information 
inconsistency and insufficient data accuracy. Therefore, a multi-sensor fusion perception system is 
proposed in this study to monitor ship motion in inland waterways. Firstly, a hardware platform of multi- 
sensor fusion ship motion perception system composed of maritime radar, AIS, cameras and other 
accessories is constructed. Secondly, by utilizing the target detection and tracking algorithms, track 
association algorithms, the ship motion data collected from the three sensors are integrated. Finally, the 
performance of the ship motion perception system is verified by field experiments in day and night. The 
experimental results indicate that the integrated ship motion perception system with multiple sensors is 
able to improve the information consistency and data accuracy of ship motion apparently in inland 
waterway compared to other perception systems. 
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1. Introduction 

Ship motion perception is of great importance in maritime supervision and port traffic management. It 
is the fundamental procedure of analyzing maritime traffic conditions, supervising the illegal behavior 
of ships and ensuring the ship traffic flow and safety in port waterways. Hence, it has always been a hot 
topic in the field of maritime traffic engineering. 

Ship motion perception data can benefit maritime study in many aspects. In the aspect of law, it can 
help legislators formulate ship traffic regulations scientifically and provide a reference for maritime 
regulators to judge the traffic status [1-3]. In the aspect of the crew, it can provide them with the empirical 
tule of ship navigation and manipulation. The uncertainty of the driver's judgment of the ship traffic state 
under the ship’s encounter state can be reduced with the help of data [4]. In the aspect of maritime 


regulator, it can be used to grasp the methods, characteristics and laws of the ship in manipulating and 


avoiding collisions at the micro level, and find the problems existing therein [5]; it can provide theoretical 
and data support for the development of navigational aids and equipment [6]. From the point of view in 
ship operation, it can be used to predict the sailing time of ships and assist in port scheduling and 
operation [7]; it can be associated with ship emissions to formulate optimal sailing routes and navigation 
plans [8]. 

In this study, a multi-sensor fusion ship motion perception system based on radar, AIS data and 
cameras is proposed for inland waterways. Compared with traditional methods using a single type of 
sensor, the proposed system can overcome the poor performances of sensing reliability and data integrity 
that existed in the traditional methods, and can collect ship navigation data more comprehensively and 
accurately in real time. The rest of the paper is organized as follows: the existing ship motion perception 
technology is reviewed first in section 2. Then, section 3 describes the configuration of the system from 
the perspectives of hardware and software in detail. As follows, section 4 presents field experiments to 
evaluate and verify the performance of the system. Finally, the potential advantages and limitations of 


the system are discussed in section 5. 


2. Related work 

Inland rivers and port channels have high traffic density, which lead to massive ship encounter happen 
and collision avoidance decision based on ships motion perception need to be made frequently. The main 
facilities used in the ship motion perception in inland rivers and port waters are maritime radar, AIS and 
cameras. 

Among the facilities, maritime radar is the first one that has been used. One challenge of detecting 
ship targets using radar is solving the problem of clutter adaptive filtering in the procedure of signal 
processing [9]. The output format of maritime radar data is the radar wave reflection images output from 
maritime radar contain massive irrelevant objects such as bridges, coastlines, islands and buoys, which 
significantly increases the difficulty of ship recognition. Ma et al. [10,11] proposed a radar target 
extraction method based on multi evidence fusion by analyzing the inherent speed, heading and position 
laws of ships in motion to increase the radar image recognition ability. He et al. [12] proposed a target 
recognition algorithm based on improved FCM according to the ship motion characteristics and the 
shadow length of the moving ship in the radar image, which performed a higher recognition accuracy 
than the traditional object detection algorithm. Nerea et al. [13] applied artificial intelligence technology 
to the detection of small and medium-sized ship targets in sea-clutter environments and designed a neural 
network training set for approaching Neyman-Pearson detector. 

AIS is a maritime communication device using the very high frequency (VHF) radio broadcasting 
system, which is helpful to realize the message transmission between ships and shore-based monitoring 
[14]. AIS data includes static information such as ship name, type, and size, as well as dynamic 
information such as ship position (longitude and latitude), speed and course. The transmission interval 
of dynamic information of shipborne AIS equipment changes according to the navigation status of the 
ship, which is usually 6 (Class A) and 10 seconds (Class B). Own to its working mechanism, AIS 


information transmission is easy to be affected by terrain [15], transmission distance [16] and ship density 


[17], resulting in error, loss, or forgery of information [18]. Thus, it is necessary to carry out error data 
identification and repair work [19,20]. With rich information, AIS data has become the main data source 
for ship behavior research [21-23]. 

The video monitoring camera can collect images within 1 km, and it is often used as a supplement to 
monitoring key water areas. It can reduce the on-site patrol frequency of maritime supervisors and save 
the response time after danger, but staffs on duty needed to stare on the video screen 24 hours. With the 
development of image processing technology, more and more image processing technologies have been 
used for ship target tracking and recognition in video images [24,25]. Cao et al. [26] proposed a ship 
recognition method based on Morphological Watershed Image Segmentation and Zemike moment. In the 
marine ship target detection, a large number of clutters in the video background significantly impact the 
detection effect. Chen et al. [27] used the characteristics of Gaussian mixture model (GMM) to judge 
whether the new pixels in the video belong to the foreground or not and used background subtraction for 
moving ship detection. With the advance of the deep neural network in image recognition, its application 
in ship target recognition is also becoming more and more popular. Huang et al. [28] proposed an image 
target detection method based on YOLOV3, and the model was trained and verified with the ship data 
set. Ren et al. [29] proposed a ship recognition method based on Hu invariant moment and convolutional 
neural network (CNN) by using the characteristics of Hu invariant moment of image. In ship motion 
perception system, not only the target needs to be detected, but also the target needs to be tracked. There 
are two kinds of traditional target tracking approaches: generative-based method and discriminative- 
based method. Common generative-based tracking algorithms include particle filter [30], mean shift [3 1], 
Kalman filter [32] and feature-based tracking. However, in the process of target tracking, the target 
features are usually similar to the background, and only modeling the tracking target can no longer meet 
the requirement of the target tracking task. Wang Naiyan proposed the DLT [33] algorithm based on deep 
learning technology in the field of target tracking, which showed that deep learning technology can 
achieve more accurate results than traditional methods in target tracking. Ma et al. [34] proposed a target 
tracking method that combines deep learning and correlation filtering, and achieved remarkable results; 
David adopted a deep learning framework in 2016, which is the first time that deep learning-based target 
tracking has achieved every 100 frames per second or more [35]. 

Although maritime radar, AIS and video monitoring cameras mentioned above have many applications, 
the accuracy and content integrity of ship navigation dynamic data sensed by only one of them is still 
insufficient. It’s hard to distinguish moving objects with sailing ships via radar images and AIS could be 
interfered by broadcast radio or buildings nearby the inland channel. In contrast, using an integrated ship 
motion perception system with multiple sensing facilities can obtain more accurate and consistent data 
based on information fusion. The Radar and AIS can work together very well because their information 
complements each other. Habtemariam et al. [36] analyzed the characteristics of AIS and radar sensor 
data and proposed a measurement level fusion algorithm of radar and AIS information based on Joint 
Probabilistic Data Association (JPDA) framework, which solved the uncertainty in AIS target tracking 
allocation through Bayesian inference. Liu et al. [37] proposed a data association algorithm based on 


multi-factor fuzzy judgment and grey correlation analysis to improve the correct correlation between AIS 


system and radar target. Cao et al. [38] used standard deviation to normalize the preselected data, then 
used a backpropagation (BP) algorithm to adjust the weight and threshold of trajectory correlation cost 
function; the proposed methods can improve the accuracy of AIS system and radar tracking. However, 
these two kinds of sensor data are not as intuitive as video surveillance images, and the verifiability of 
fusion results is not good neither. 

The ship motion perception system works together from the most primitive manual method to the 
current AIS system, radar, electronic chart and other technical methods, which greatly improves the 
safety of ships and the level of collaborative management of ships. Recently, the ship motion perception 
system based on computer vision is an effective supplement to the existing system. However, the existing 
research is mostly based on the SAR ship data set, which is different from the application scenarios of 
ship motion perception. Target tracking is a vital part of ship motion perception. The method has 
developed from traditional Kalman filtering, Mean shift, etc. to correlation filtering and deep learning 
methods. For ship target tracking, the sea surface conditions are complex, and there are problems about 
target occlusion and ambiguity, which are difficult to solve by traditional methods. Additionally, there 
are relatively few researches on the fusion of vision and other ship motion perception, and the research 
on information fusion of AIS, radar, and electronic chart is still the mainstream. How to associate, match 
and fuse the multi-target information of visual perception with radar and AIS-perceived multi-target 
information still needs to be further explored. 

Aiming at the demands of ship motion perception in inland river and port waters, this research adds 
camera as an information source on the basis of radar and AIS data. The addition of cameras can provide 
new continuous tracking information of ship targets for supervisors. Moreover, by utilizing the target 
detection and tracking algorithms, the ship motion data collected from the three sensors are integrated to 
strengthen the accuracy of information fusion, which is helpful for supervisors of inland waterways to 


verify the type of ship, construct a closed-loop of tracking and make decisions. 


3. System configuration 

The configuration of multi-sensor information fusion system is described in two parts: hardware and 
software. Hardware including AIS, Radar and Cameras modules are introduced in Section 3.1. And 
software algorithms using for ship motion recognizing, tracking and fusion on the basis of above 
hardware module are illustrated in section 3.2. 
3.1. Hardware 

As shown in Figure 1, the multi-sensor ship motion perception system proposed in this paper mainly 
consists of a CWFM Solid-state Radar (SIMRAD, 4G), an AIS terminal (ZYE, ZY-1000), a camera 
(Hikvision, DS-2CD6A64), an industrial computer for data processing and so on. Among them, the radar 
is to obtain the radar echo image data of the surrounding navigation environment and ships in real-time. 
The AIS receiver receives the AIS message broadcast by the surrounding ships and obtains navigation 
information such as the positioning, speed and navigation of the sailing ship. The camera collects video 
information of nearby ships to identify ship type, azimuth, distance, etc. The computer is responsible for 


collecting, processing and fusing the data output by radar, AIS, video camera and other sensors. The 


modules of radar, AIS and camera are introduced in detail as follows. 
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Figure 1. Photo of the proposed multi-sensor ship motion perception system. 

(1) Radar module 

The system uses a SIMRAD 4G FM CW Solid-state Radar, and its detection range is between 1/32 to 
36 nm. Doppler beam sharpening is used to improve the radar resolution. At the same time, FM 
continuous wave technology is also used to improve the accuracy of distance information measurement. 
Finally, high-precision perception and recognition are achieved for the range, azimuth, motion speed and 
other parameters of ship targets within the monitoring area. The data output format of radar is the picture, 
and the data interval is 2 seconds. 
(2) AIS module 

The AIS mode used in the system meets the relevant standards of AIS Class B. It can receive data 
related to ship navigation safety in real-time, and provide relevant AIS data for port and shore 
management, ship warning, collision avoidance, and ship supervision. The data output format of AIS 
equipment is text, and the data interval is from 30 seconds to 3 minutes. 
(3) Camera module 

The system adopts Hikvision 24 million ring eagle eye DS-2CD6A64 series camera. The camera 
consists of four horizontally spliced camera modules, formed a viewing angle of 180° in horizontal and 
80° in vertical. The four camera modules are hoisted at the lower end of the holder, which can rotate with 
the holder in the horizontal direction of -90°~90°, and swing with the holder in the vertical direction of 
-45°~90°. For sake of the ultra-wide viewing angle, the camera is precisely suitable for image monitoring 
in a wide water area. The data output format of the camera is the video frame, and the frame interval is 


1/60 seconds. 


3.2. Software 


The software algorithm of the multi-sensor information fusion method adopts a distributed architecture 
shown in Figure 2. The algorithm is mainly composed of four modules. The first is the image recognition 
and tracking module, which recognizes and classifies the ship image, tracks it, and outputs the image 
target features. The second is AIS data processing module, which analyzes and denoises the ship's AIS 
information, subcontracts the data according to MMSI code, and outputs the AIS target motion 
characteristics. The third is the radar target recognition and tracking module, which outputs the target 
characteristics of radar images. The fourth part is the target information fusion module, which outputs 
the target fusion features after data registration, fuzzy evaluation and track correlation. Software 


algorithm are demonstrated in detail as follows. 
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Figure 2. Software algorithm. 

3.2.1. Target recognition 

The target recognition system is described in two parts: ship image recognition and radar image 
recognition. 
(1) Ship image target recognition 

The ship image target recognition is responsible for obtaining the pixel position, area size, target 
reliability and target classification of ship image target, and it adopts Darknet network model based on 
YOLOv3 deep learning framework algorithm. Darknet deep learning framework is an open-source neural 
network framework proposed by Joseph Redmon, and YOLO (You Only Look Once) is the core target 
detection algorithm for this framework. In the YOLO algorithm, the object detection problem is treated 
as a regression problem. The bounding box and the probability of category can be directly predicted from 
the input image with a convolutional neural network structure. This method improves the basic 
classification network structure and the two classification prediction method of targets in traditional deep 
learning, and realizes ship tracking and ship type recognition. The real-time ship tracking and recognition 
algorithm based on Darknet network and YOLOv3 algorithm achieves the balance of speed and accuracy. 
The flow chart of the algorithm is shown in Figure 3. 
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Figure 3. Ship image target recognition algorithm. 
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(2) Ship radar target recognition 

The radar target recognition module in inland river and port waters firstly sets the capture area, 
eliminates the coastline and navigation buildings, and retains the accessible area of ships only. When the 
position of the target falls into the capture area, it is captured, and the moving target ship is detected by 
image processing methods such as background difference method, image morphological processing and 


median filtering. 
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Figure 4. M/N logic principle of sliding window method. 

The process of target acquisition is an essential part of the process of track initiation. The track 
initiation algorithm based on M/N logic shown in Figure 4 is adopted. In N consecutive scanning cycles, 
when the number of detections meeting the threshold conditions in the time window reaches a specific 
value M, the track initiation is successful; otherwise, keep the number of time windows unchanged and 
move one scanning time in the direction of time increases. 

3.2.2. Target tracking 

The target tracking process is composed of ship AIS trajectory tracking, ship radar track tracking, ship 

image target tracking, and their details are presented as follows. 
(1) Ship AIS trajectory tracking 
According to Maritime Mobile Service Identities (MMSI), the navigation track data of the same ship 


should be subcontracted. For discontinuous navigation ships such as ferries and official ships, the tracks 


of the same MMSI code with a time interval greater than a certain threshold value shall be segmented to 
ensure the continuous and stable navigation data of each ship section. Finally, the single-day ship 
trajectory data can be extracted. 
(2) Ship radar track tracking 

Ship radar target tracking mode mainly includes target acquisition, track correlation, filtering, and its 
specific process is shown in Figure 5. By analyzing radar image, the current GPS coordinate position of 
the radar and the inertial navigation information of the current radar can be established. The information 
of radar targets is also extracted, which includes the heading and speed of the target relative to the radar, 
the position information of the target and the size estimation information of the target. With the target 


tracking, the radar image data is acquired. 
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Figure 5. Radar target recognition and tracking. 

The nearest neighbor method is adopted in the process of radar tracking. By judging the predicted 
value of the target output by the filter, the wave gate is established with the predicted value as the center, 
and the target is collected within the wave gate range to judge whether the target and track are related. 
Under special circumstances such as target intersection, bridge crossing and being blocked by navigation 
buildings, it is difficult to identify the radar target. Therefore, the position and velocity state of the target 
at the next time are calculated through the motion equation. 

(3) Ship image target tracking 

The Kalman filter and Hungarian matching algorithm are adopted in ship image target tracking. 
Through the tracking of ship image target, the track correlation of continuous multi frame image target 
is carried out to obtain the continuous track of ship image target. The main steps of the algorithm are as 


follows: 


© Creation of tracker. Detect the ship image at the beginning, initialize the detected target, create a new 
tracker to track it, and mark the serial number of the target. 
© Target matching. The target frame obtained from the initial frame detection can be used to obtain 
covariance prediction and state prediction through Kalman filter. Taking the intersection over union (IOU) 
as the matching basis, the IOU value of the prediction of all targets and the detection frame was calculated 
at first; then, the maximum and unique matching through the Hungarian algorithm was obtained; finally, 
the matching pair whose matching value is less than the threshold was deleted. 
®© Status update. For the target without matching, a new tracker is established, and the target detection 
frame of this frame is used to update the Kalman filter. The updated contents include Kalman gain, state 
matrix and covariance matrix. The tracking frame of this frame is the state update value. 
3.2.3. Data fusion 

In the Integrated Ship Motion Perception system of ship navigation in inland river and port waters, the 
main perception method will still be AIS and radar. Benefit from their large-scale perception abilities and 
having a variety of perception means, using data fusion technology and integrating a variety of perception 
technologies can complement and expand each other. In this paper, AIS, radar and camera are integrated 


to dynamically perceive the ship navigation status. The specific model is as shown in Figure 6. 
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Figure 6. Structural model of multi-sensor information fusion. 

(1) Data space and time registration 

As shown in figure 6, the output data of the radar, AIS and camera have different data formats and 
coordinates. The radar targets were expressed by polar coordinates, the AIS data uses latitude and 
longitude coordinate system, and the camera uses pixel coordinates. In data space registration: AIS and 
video image fusion need a unified position coordinate system. In this paper, AIS coordinates and radar 
polar coordinates are converted into camera pixel coordinates through coordinate conversion to unify 
with the ship image target in the same coordinate system for track correlation. The ideal AIS ship image 
coordinates are calculated by using the ideal pinhole imaging model. However, the lens of the optical 


camera is distorted (geometric distortion of the image), so the ideal AIS ship image coordinates were 


converted into the image coordinates in the real distorted image after calculation. The polar coordinate 
system is adopted for the radar coordinates, and the radar data is inversely transformed by three- 
dimensional coordinates to complete the spatial matching between the target information and the visual 
image. 

In terms of data time registration: since the sending time interval of ship AIS data ranges from 10 
seconds to 3 minutes and the update frequency of image data is 25 Hz, it is necessary to predict the AIS 
space-time trajectory during space-time registration, so as to align the time of ship AIS data involved in 
fusion with image data. In this paper, the AIS ship is assumed to move at a uniform speed. Using the 
characteristic values of longitude, latitude, speed, heading and steering rate of AIS ship target and taking 
1 second as the cycle, the cubic spline interpolation method is used to predict the AIS space-time 
trajectory in a short time. 

(2) Track correlation 

In this paper, the comprehensive factor fuzzy evaluation method is used to determine the correlation 
between AIS, radar and image target tracking, so as to determine the correlation of each sensor target, 
which mainly includes three parts: coarse correlation, fine correlation and comprehensive judgment. The 


track correlation framework of comprehensive factor fuzzy evaluation is shown in Figure 7. 
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Figure 7. Comprehensive factor fuzzy evaluation track correlation framework. 
a) Coarse correlation 


Coarse correlation is to find the target subset satisfying Eqs. (1) in the current AIS and radar target set 


according to the pixel coordinates (U,V) of the midpoint at the bottom of the target in the ship image. 
Uy E U| <T, 
[Vy T V, | < I 


(1) 


Uy Vy) 


where is the coordinate point of AIS and radar target image, and T, ; T, are pixel threshold. 


b) Fine correlation 
The ship image target, AIS data and radar target meeting the condition of rough correlation are 
smoothly correlated by a comprehensive factor fuzzy evaluation algorithm. Smooth correlation mainly 
includes determining fuzzy factor set, constructing membership function model, comprehensive 
evaluation and calculating compactness. 
© Fuzzy factor set 
In this paper, pixel abscissa, pixel abscissa change rate and pixel ordinate are selected as the fuzzy 


factor set, and the calculation formula is expressed as: 
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where (u;,v;) is the target pixel point of the ship image, (uo; vo) is the optical center pixel point of the 


image, f is the focal length of the camera, d, is the physical size of a single pixel around x axis, h is 


the installation height of the camera, OAi is the angle between the AIS target and ship’s heading at the i- 


th time, rAi is the distance between the target object and the camera and ” is the physical size of a 


single pixel around y axis. 
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Figure 8. The probability density function and distribution diagram. 

In this paper, the probability density is estimated by using the training sample data of U1, U2 and U3. 
The probability density functions corresponding to subordinate functions S1, S2 and S3 are obtained, 
and then the probability density function is normalized. 

First, as shown in Figure 8, the probability density functions of S1 and S3 adopt semi normal 
distribution, and the probability density function is estimated according to the data samples of U1 and 
U3, while the probability density function of S2 adopts exponential distribution to estimate the 
probability density of U3. Secondly, the probability density functions of S1, S2 and S3 are normalized 


as follows: 
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where subordinate functions S1, S2 and S3 vary between [0,1]. 0 indicates relevance and 1 indicates 
irrelevance. 
c) Comprehensive evaluation 


Firstly, the judgment matrix g; used in this paper is expressed as: 


&i =} a, i (4) 


where i is the membership grade calculated according to the subordinate function Si. The 
coefficient @ is determined by the membership degree calculated by the training set samples, and its 


relationship is expressed as: 
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a, =0.374 a, =0.267 a, =0.359 


According to the statistics of training samples, 


Secondly, the continuous multiple of the ship image target and the corresponding single AIS target are 
averaged. Then, the tightness coefficient takes the maximum value of multiple different AIS targets at 


the same time [9]. 


si = arg max È} g; 


(7) 
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j <T., judgment irrelevant 8) 
where r is the relevant decision threshold, and it is taken as 0.6. 
3.2.4 Quality management 
The track correlation quality is expressed as: 
qij (n)= lij (n—1)+ Aq; ; (n) 0) 
1, judgment relevant 
Aq, (7) = -1, judgment irrelevant 
ese (10) 
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where When , the two tracks are fixedly correlated. 


3.2.5 Target information fusion 
In the process of radar and AIS data fusion, due to the high accuracy requirement of radar, the 
coordinate information is calculated based on the radar measurement data and the correlation between 


longitude, latitude and distance. The fusion of AIS and radar targets is realized after the process of fusion. 


For the related target, the information is fused to obtain the fused target. AIS target information, radar 
target information and target information after fusion are shown in Table 1. 


Table 1. Target information fusion of AIS and radar. 


AIS target Radar target Fusion target 
MMSI distance MMSI 
position relative orientation time 

UTC time time distance 


track direction 
speed 
ship heading 


steering rate 


relative ship speed 


relative orientation 
speed 
relative ship speed 


longitude and latitude 


steering rate 
track direction 


Table 2. Video target information fusion. 


AIS target Video target Fusion target 
MMSI relative orientation MMSI 

time target detection box time 
distance target category distance 


relative orientation relative orientation 


speed speed 

relative ship speed relative ship speed 
longitude and latitude longitude and latitude 
steering rate steering rate 

track direction track direction 


The fusion result of video tracking target information and radar AIS target information is shown in 
Table 2. In the matching process, the image target has the highest priority. When there is only one image 
target, the target is retained. At the same time, the image is helpful to judge the absence of AIS target. 
Ship image information includes relative azimuth, relative distance, target frame coordinates, size and 
other information; AIS information includes ship name, MMSI, longitude and latitude, speed, heading, 
steering rate, etc; the target information after merging includes the target ship name, MMSI, longitude 
and latitude, speed, heading, heading, steering rate, relative azimuth, relative distance, target frame 
coordinates and size, etc. After fusion, the data with more information is acquired, so the positioning 


accuracy is improved. 


4. System verification 
The system verification of multi-sensor information fusion system is conducted by field experiments 
in this section. Firstly, the overall design of ship motion perception system software in inland river is 


demonstrated in section 4.1. Then, effect of image fusion and radar fusion is verified separately in section 


4.2 and 4.3. Lastly, on contrast, multi-source fusion verification is carried out under various practical 
scenes. 
4.1. System demonstrator 

The overall design of ship motion perception system software is demonstrated in Figure 9. As indicated 
in the function architecture shown in Figure 9(a), the software consists of three parts: control and 
acquisition, perception and information display. The main function design follows the content of demand 
analysis. The perception part mainly implements the algorithm content studied in this paper. The 
information display part is designed for video preview, which is used to show the real-time perception 
effect of images. The interface of the information display is presented in Figure 9(b). The electronic River 


map can display AIS targets, superimposes the radar map, and the fusion perception effect of radar and 
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(b) User interface. 
Figure 9. Ship motion perception system software. 
The interfaces of each part of the software are as follows: 
(1) AIS module 


The AIS message information is displayed at the center of the interface. The AIS message contains 


dynamic information such as ship position, heading, speed, longitude and latitude, steering rate and static 
information such as ship name, call sign and ship type. 
(2) Radar module 

The main operation interface is divided into several different functional areas. The user can connect 
the radar, set the frequency of image storage and other operations. 
(3) Camera module 

The camera module is set to control the connection of the camera and the storage of images and videos. 
At the same time, it can preview the video and display the target list. 
4.2. Image fusion verification 
(1) Optimized YOLOv3 network structure description and algorithm evaluation 

The data set used for the accuracy verification of ship target detection is verified by 10-fold cross- 
validation. At the same time, the mean average precision (mAP) commonly used in target detection task 
is adopted as the evaluation index of detection. The test set is tested and the average value is taken after 
ten times. Generally, when evaluating the performance of the target detector, it is qualified if IOU 
between the detection frame and the actual target frame is greater than 0.5. The overall mAP of the target 
detection and verification result of the system reaches about 93.04%. Considering the high demand for 
target positioning in environment aware applications, the mAP values based on different IOU thresholds 


were tested and compared before and after improvement. 
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Figure 10. Comparison of evaluation results. 

As it can be seen from Figure 10, the system optimized and adjusted the original loss function and 
network super parameters of YOLOv3, which improved the positioning accuracy. With the improvement 
of the positioning accuracy requirements, the mAP value decreases accordingly. When the IOU is 
increased to about 0.6, the algorithm can still yield an mAP value of about 0.9, while the improved 
YOLOv3 is slightly improved in terms of the mAP value in general. 

(2) Continuous evaluation of ship target tracking 

The system uses the optimized YOLOv3 model as the detector. By introducing the depth feature and 
cascade matching algorithm, the tracking algorithm performed very well in the tracking continuity of 
ship images. After introducing the depth feature information, the tracker can still track the target correctly 


under the occlusion for a long time, so as to reduce the occurrence of ID switching. In order to reduce 


the false alarm caused by YOLOv3 detector, the detection confidence threshold was appropriately 
adjusted, and the final tracking result performance was greatly improved. 
(3) Fusion accuracy evaluation 

After the system detects and tracks the ship target, it will finally fuse with other multi-source 
information. An output result of the ship target fusion experiment is shown in Figure 11. The nine-digit 
code of MMSI, speed and heading information of the ship are displayed on the picture, and the historical 


fusion track of the ship is also displayed at the same time. 
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Figure 11. Effect of ship target recognition, tracking and fusion. 


After fusion, the output data of the system is described in the following table: ID is the identification 
number of the system to the ship, and each new identification target ID value is added by 1. The size of 
the target identification frame is marked as BBOX, and the abscissa and ordinate of the midpoint pixel 
at the bottom of the target identification frame are marked as BCX and BCY, respectively. 


Table 3. Ship image target data. 


ID BBOX BCX BCY 
'103' [280,547,153,49] 356.5 596 
'103' [273,546,166,49] 356 595 
'103' [304,555,132,41] 370 596 
'103' [314,554,132,41] 380 595 
'103' [325,554,132,41] 391 595 
'103' [335,554,132,41] 401 595 
'103' [349,544,159,47] 428.5 591 
'103' [360,543,170,48] 445 591 
'103' [374,542,170,48] 459 590 
'103' [381,540,167,46] 464.5 586 
'103' [382,541,175,47] 469.5 588 
'103' [456,540,120,45] 516 585 
'103' [433,538,160,46] 513 584 
'103' [373,541,269,44] 507.5 585 
'103' [477,538,126,46] 540 584 


The final video fusion accuracy of the system is shown in Figure 12. When the threshold which applied 
to compared spatial difference between ships in AIS and image is around 2 °, the accuracy tends to be 


the largest, and the overall variation range with the threshold is small, mainly because the image is less 


sensitive to orientation. The maximum error rate is about 3%, which is mainly limited by the accuracy 
of image recognition, and the whole is still at a low level. The result of video perception is relatively 
accurate, but the target will still be lost due to occlusion during the ship meeting; After fusion, the 


perception result is good, but there are still few targets that cannot be perceived due to occlusion. 
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Figure 12. Video fusion effect. 
4.3. Radar fusion verification 
(1) Data introduction 
Figure 13 presents the original radar and AIS data acquired on the Yangtze River, Wuhan, China. The 
radar mode can form a 1024 x 1024 pixel image in 2.5 seconds. The coastline in the radar image was 


removed. The AIS data was irregular in time, so it needed interpolation. 


Figure 13. Original data sample. 
(2) Evaluation parameters 
In the actual application of the equipment, the visible range of the radar is taken as the sensing range. 
When a target passes through the sensing range, if the target is correctly identified and the effective 
tracking time reaches more than 80%, it is considered that the target is correctly perceived. In contrast, 
all the perceived targets except for the correct ones are considered as misperceived targets. To evaluate 
the performance of the perception algorithm, Accuracy and ErrorRate are designated as the evaluation 


parameters of the experimental results, and their expressions are given as follows 


Accuracy = 


(11) 


ErrorRate = 


zla S|2 


(12) 


where T represents the number of correctly perceived targets, E represents the number of incorrectly 
perceived targets, and M represents all actual targets. In the experiment, if there are many missed 
detection, the number of perceived targets will be less than the actual number, so the sum of accuracy 
and error rate will be less than 1; if there are many false detections, the perceived target number will be 
more than the actual target number, which leads to the sum of accuracy and error rate greater than 1. 
(3) Experimental results 

By adjusting the threshold value of AIS data and radar fusion, the changing trend of Accuracy and 
ErrorRate are shown in Figure 14. When the threshold is about 100m, the accuracy, indicating that most 
matching pairs are within 100m. The maximum value of accuracy is about 93%. When the threshold is 
set less than 50, the effect is poor with small accuracy values, which denotes that the coordinate error 
between AIS and radar target detection is relatively large, so it is difficult to match the target with a small 
threshold. It is observed that the errorRate increases with the increase of the threshold, and its maximum 
value is about 7%, which is mainly due to the matching between some drifting AIS targets and radar 


jamming targets when the threshold is too large. 


After data fusion, the target detection effect is shown in Figure 15. The triangles are the AIS detection 
targets, the circles are the radar detection targets, and the squares are the fused targets. It is seen that the 


target in AIS and radar match rate has been greatly improved after fusion. 
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Figure 14. Fusion effect of AIS and radar. Figure 15. AIS and radar data fusion. 

4.4 Multi-source fusion verification 

In this section, the evaluation of the accuracy of the multi-source fusion algorithm under single-target 
and multi-target scenes are presented, respectively. 
1) single-target scene 

A ferry ship named Jiangcheng No.5 ship was chosen as the single-track target. The ship travelled 
from Wuhanguan wharf to Zhonghua Road wharf in Wuhan, China, and its real track is recorded by a 
high-precision differential GPS device (decimeter level) equipped on the ship. As shown in Figure 16, 
the recorded real track of the ship is composed of two turns and three stages of constant speed and a 
relatively stable course. It is noted that the X and Y axials in the figure denotes longitude and latitude, 


respectively. 
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Figure 16. Real track of ferry ship. 

The tracking performances of the methods, including AIS track, radar track, video and radar fusion 
track, and the fusion track of the proposed multi-sensor fusion ship motion perception system, were 
evaluated. After aligning the time points of data acquired by all the tracking methods and converting the 
coordinates of the data to the geodetic coordinate system, the performances of all the track methods were 
compared, as shown in Figure 17. In which lines with different colors denote the ship’s trajectory tracked 


by various sensor separately and integrally. 
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Figure 17. Various sensor tracks and fusion tracks. 

The camera is accurate for angle measurement, but its performance in distance measurement is poor, 
hence this paper uses radar to measure distance and camera to measure angle information to realise a 
multi-source fusion track. The final fusion track is closer to the real track of the ship target than other 
tracks, which can prove that the target fusion algorithm is effective in Figure 18. The relative errors of 
all track methods in terms of longitude and latitude in a sampling period of 120 s are presented in Figure 
18. It is observed that the data of fusion has much smaller errors than other track methods, indicating the 


advantage of the target fusion algorithm. 
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Figure 18. Fusion error diagram. 


2) multi-target scene 


In order to verify the detection and tracking effect of the system in the multi-target ship scene, tests 
were carried out in different scenes during the day and night. The test environment is shown in Figure 
19. Two data groups lasting one hour acquired during the day and night respectively were analyzed, and 
the comparison of ship recognition and tracking performance of single sensor and multi-sensor data 
fusion methods are compared in Table 3. Table 3 reveals that, given a period of 1 h, the total number of 
ships at night and day is 443 and 1368 relatively. It is found that the day and night have little impact on 
radar which detection rate is 70.32 and 69.30 respectively. However, the perception ability of video ship 


targets at night decreases greatly, and the recognition rate decreases from 68.34% to 57.33%; Among the 
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three sensing methods, the fusion sensing target recognition rate is the highest, reaching 72.15%. 
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Figure 19. Multi-objective test environment. 


Table 3. Ship identification rate. 


Index Sensor Day Night 


Il Radar 962 307 
I2 Video 935 254 
13 Fusion 987 321 
14 All ships 1368 443 
15 11/14(%) 70.32 69.30 
16 12/14(%) 68.34 57.33 
I7 13/14(%) 72.15 72.46 


5. Conclusion and Discussions 

This paper designed an integrated ship motion perception system based on muti-sensors information 
fusion. In order to evaluate the reliability, consistency and accuracy of the system, a pilot test was carried 
out in Wuhan waterway of the Yangtze River. Compared with the existing ship motion perception 
methods, it has the following main advantages. 

In terms of hardware, the system proposed and designed in this paper forms a three-layer monitoring 
mode which can cover inland waterways within 1.5 nautical miles. The accurate trajectory of the ship in 
the waters is obtained through the information fusion of radar and AIS, and the video image is used to 
verify the ship motion and ship type. Consequently, the system has better robustness and reliability in 
ship motion perception. 

In terms of software, the system integrates three ship motion perception techniques: radar, AIS and 
camera. In the fusion of radar, AIS and camera, the spatial registration of three kinds of sensor data is 
realized through coordinate transformation. Then, the track correlation between AIS, radar and image 
target are carried out by using the comprehensive factor fuzzy evaluation method. After the track 
correlation is completed, the redundant information in the three-sensor data is eliminated to realize the 
comprehensive, consistent, accurate and real-time monitoring of ship motion. 

The ship motion perception system can also be deployed and applied to the smart ships. Although the 
system has been preliminarily verified, the wide application of the device is still in the early stage. The 
next research work will focus on improving its performance in various experimental and practical 
environments. And the structural design will be optimized under considering the factors of waterproof 
and dustproof. Moreover, introducing the correction device of longitude and latitude position and attitude, 


the system can also be deployed on the intelligent ship as the situation awareness sensing system. 
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